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LINE OBJECT SCENE GENERATION APPARATUS 

Field of the Invention 

This invention is directed generally to systems which monitor and record 
motion events. More particularly, it relates to the accurate management and control of time- 
sequential imaging and display, with application in numerous fields of science and 
technology. Most particularly, the invention provides a total race-management system which 
has wide ranging utility in measuring timed sporting events. 

The invention also relates to systems and methods for generating a scene by 
compilating successively-scanned line objects. As such, the invention also concerns apparatus 
for compressing those scenes into manageable data files and for color coding scene 
information efficiently. 

Background of the Invention 

Prior art systems which track and record a motion event over time are 
overwhelmingly directed towards the support of racing events. Standard photographic 
techniques which monitor the finish line of a race are known. Typically, cameras equipped 
for high resolution imaging view the finish line and sequentially capture pictures at a high 
rate for later use by an interpreter. However, this process is cumbersome, wasteful, and time- 
consuming, requiring, for example, an apparatus of photographic film and paper, processing 
chemicals, and image enlargers or projection optics. Consequently, most races rely on human 
judges and revert to ,f photo-finish M technology only in extremely close races. Specialty 
Instrument Corporation provides a myriad of such electronic and photo-finish systems under 
the trademark Accutrack™. U.S. Patent No. 3,829,869 exemplifies one such Accutrack™ 
system. 

Because of the problems with the "photo-finish" technology, numerous other 
systems for monitoring racing events have been developed. However, these >other methods 
and systems for timing sporting events present new difficulties. Video systems which record 
and display races in a standard television or video format are popular, but regardless of the 
particular implementation of these systems, a portion of the electronic image remains on an 
analog medium, e.g., video tape. Since analog data from the systems consists of a continuum 
of information over time, it is relatively difficult to accurately apportion to a unique time 
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interval. It is even more difficult to access a particular moment in time in the recorded 
sequence because the associated system must search the storage medium, typically having a 
long physical length in a spooled format, e.g., a video cassette. This presents both limitations 
and difficulties for users wishing to simultaneously record, view the current race, and review 
earlier segments of the race (or even a previous race) because only one user can have access 
to any of the information stored and recorded at any one time. 

A further difficulty in analog data is that it must be converted to a signal 
usable for video, television, or a computer before it is displayed. For example, after a 
completed search, the selected video tape segment is typically sent to active memory before it 
can be processed by a computer and, quite possibly, by supplemental complex graphics 
generators. Altogether, the analog format and related processing adds to the time required to 
review a race and therefore lengthens the decision making process. 

Another problem faced by race systems occurs in the management of extended 
time events, like a marathon or bicycle race, which can last for hours or until each entrant 
finishes. The runners or cyclists cross the finish line in groups; and for long periods, the 
finish line is void of persons. The relevant information at the finish line is thus sporadic, and 
includes significant amounts of "dead" time. In analog systems, this dead time is 
nevertheless recorded and stored so that the system can retain time synchronism with the 
event, even though it is generally useless for other reasons and adds to the time required for 
processing and reviewing the race. 

Several race systems have attempted to improve the management and 
accessibility of data taken during a race by transforming the recorded information to a digital 
equivalent. But, these systems also revert to an analog format before displaying the race on a 
screen. As examples, U.S. Patent No. 4,797,751 shows a video recording system having both 
digital and analog sections to provide display on a common cathode ray tube (CRT). U.S. 
Patent No. 5,136,283 similarly describes another partially digital system which displays races 
on a standard television format. These analog/digital systems still have many of the problems 
inherent in all analog systems. 

It is, accordingly, an object of the invention to provide a system for recording 
and displaying a time-sequential scene of bodies crossing a plane. In particular, the system 
provides improvements in managing and recording timed sporting events which reference 
bodies or entrants crossing a finish line relative to the start of an external event. 
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Still another object of the invention is to provide improvements in the 
manipulation of timed information representative of bodies passing a plane in space, such as 
person crossing a finish line in a race. 

Yet another object is to provide improved access, control and storage of a 
scene comprising a sequence of discrete time images. 

Most particularly, it is an object of the invention to provide a race monitoring 
and recording system which can record and display a race in a digital format. 

Still another object of the invention is to store and display color scenes on a 
computer system with efficient color coding. 

These and other objects will become apparent in the description below. 
Summary of the Invention 

The invention features, in one aspect, a system for recording and displaying a 
time sequential scene of bodies moving across a plane in space. The system includes at least 
one digital camera which views and images a line object in the plane of interest. The camera 
time-sequentially captures the line object by imaging it onto an array of detector elements and 
converts the sampled signal into a digital image, or frame, of the line object. Each digital 
image frame uniquely represents a slice of the moving scene at a moment in time. The 
system also includes an image timer, with a timer processor, that responds to a preselected 
digital value from the camera and marks each frame with a digital time reference using a 
preselected number of bytes within the frame information. The image timer may also store 
the digital frames from the camera in an internal buffer. The system further includes a main 
control computer having an internal memory, a user console, and a graphics display monitor. 
The computer stores the frames from the image timer buffer as blocks of information in its 
internal memory, via an associated software pointer, and selectively displays a portion of the 
stored frames as a time-sequential scene on the monitor. A user at the computer console can 
command a variety of functions provided by the invention to manipulate and analyze the 
captured scene, most particularly to display any portion of the scene of bodies moving across 
the plane and access an associated time for any frame within. 
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The system thus summarized is particularly useful in recording and managing 
the official times of objects or entrants crossing a finish line, and at a rate which is over fifty 
times faster (and more accurate) than a related video race management system. A user can 
record and display the bodies crossing the plane of interest, i.e., the finish line, with accuracy 
while maintaining the ability to review and edit in real-time the stored images. Accordingly, 
it is useful that the time references associated with each frame are triggered with a start 
sensor and correlated to the start of an external event, e.g., a gun start signal. The invention 
thus provides, in another aspect, a timer clock that is synchronized to the start of an event and 
which provides the timer processor with an accurate reference to mark the respective 
moments in time for each frame. 

In other aspects, the digital camera can include a line scan charge coupled 
device which forms the array of detector elements. The camera can also include a greyscale 
gain controller to adjust the digital output signal according to a preselected gain level, 
preferably selectable at the main control computer, and, preferably, to a gain level 
corresponding to the digital values in the captured frames. The gain controller can function 
in a real-time fashion by adjusting the greyscale gain applied to each frame as captured by the 
camera during operation of the invention. The camera is completely computer controlled 
from a remote location. This computer control, which is achieved by passing signals along 
the coaxial cable, allows remote control of focus, zoom, pan and all other camera functions. 

In still other aspects, the resolution of the scene as captured by the camera, or 
as displayed on the screen, is selectable by a user. With respect to the camera, the resolution 
in the time-domain, i.e., in the direction of motion, is adjustable by selecting the frame rate at 
which the camera captures the digital image frames. The resolution in the spatial-domain, 
i.e., along the line object length, is adjustable by changing the camera density control which 
activates only particular detector elements in the array. 

With respect to the resolution as displayed on the monitor, the user can, in 
another aspect, zoom a particular scene in or out on the screen. For example, by zooming the 
scene out, the whole race can be viewed at once; and by zooming in, particular areas of the 
scene are enlarged on the screen, suitable, for example, to interpret the number identifier of a 
particular runner. The zoom capability is available to users at the main control computer in 
either screen dimension, i.e., in the time and spatial directions, concurrently or independently. 
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A user of the system thus described has a variety of controls at the main 
control computer. Any frame of a displayed scene can be removed, either temporarily or 
permanently, according to other aspects of the invention by "cropping" the frames. A list of 
"cropped" frames is placed into a listing memory. A time-crop control allows a user to cut- 
5 out, or "crop", uninteresting portions of the scene - for example periods of time containing no 
activity or bodies crossing the plane - while retaining an absolute time reference for the 
remaining frames, which collectively display a continuous scene. The time-crop control 
further can restore the cropped frames, by deleting them from the listing memory, or 
permanently erasing them. 

10 

In another aspect, a user can also selectively reverse the time-sequential order 
of the displayed scene so that the scene as displayed appears as though the motion of bodies 
passing the plane in space occurred in the other direction. In addition, a user can point to 
particular bodies on the display to provide both an object identifier, if selected, and a unique 
15 time identification representative of the point selected. 

The system constructed in accordance with the invention also provides, in 
another aspect, a virtual memory subsystem, like a hard-disc drive. The main control 
computer stores blocks of information into the virtual memory subsystem to free space in its 

20 own internal memory and to provide a storage medium for previous scenes, for example 
previous races. Storage into the virtual memory subsystem is initiated by an optional 
command or can occur automatically when a predetermined selectable fraction of the internal 
memory is utilized. The storage arrangement on the virtual memory subsystem is, in another 
aspect, ordered so that the main control computer can access and selectively retrieve a block 

25 of information from the virtual memory subsystem for storage in its internal memory by 

computing an offset from the initial memory location where the blocks of data are stored. In 
this manner, the internal memory of the main control computer can function as a cache for 
the virtual memory subsystem, thereby storing only a few active blocks in volatile RAM. 

30 To aid the storage capability of the virtual memory subsystem, the invention 

accordingly provides a compression system to compress the blocks of information into less 
memory space. The compression system is selectively controlled, both in initiating the 
compression of certain data and in regulating the accuracy of the compressed information. 

35 The compression system takes advantage of the fact that each digital image 

frame comprises a column of n-bit numbers, and a sequence of digital frames thus forms an 
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array of rows of n-bit numbers. In a preferred aspect, the compression occurs by first 
converting the array of digital image frames to an array of rows of m-bit greyscale numbers 
(where the integer m is less than the integer n). The converted array of rows of m-bit digital 
image frames is then reformatted in a row-by-row fashion by collecting adjacent and equal 
m-bit numbers into a group and representing the collection as a "count 11 and a greyscale 
"value". These rows are reformatted again into a sequential memory string thereby 
compressing the data to a smaller amount. Preferably, the "count 11 is either a byte or a 3-bit 
number, and the m-bit "value" is formed by a 5-bit representation of each of the original n-bit 
numbers, although the accuracy of the compression is selectable by a user by changing the 
number m. If the count is a 3-bit number, the count and value form one byte. For example, a 
4-bit number can be used to compress the data further. 

In yet another aspect, the invention can include a plurality of digital cameras, 
each with an associated buffer within the image timer, to independently capture a sequence of 
digital image frames. Thus multiple scenes are generated, preferably of a view containing 
substantially the same line object, for display on the computer. At least two scenes can be 
shown simultaneously on a single monitor from two separate cameras in both a real-time 
display or from previously recorded segments. In another aspect, one or more additional 
computers are installed in communication with the virtual memory subsystem to access and 
separately display and manipulate data captured by any one of the connected cameras. Thus, 
a second user can analyze previously recorded motion segments while a first user 
concentrates on a current motion event. 

The digital camera and image timer each have associated processing CPUs 
which can selectively compress data before transmission along a signal line. For example, 
the digital camera can reduce the bandwidth requirements of the signal line or cabling 
between it and the image timer by commanding a first compression on the digital data 
transmitted from the camera. The image timer can reduce the bandwidth requirements of the 
cabling or signal line between it and the main control computer by commanding a second 
compression on the data transmitted between the two using a similar compression scheme. 

In a preferred aspect, a single cabling is used between the image timer and 
camera. This cable preferably is in the form of a single coaxial cable that functions as a 
signal line to command various functions at the camera, a data transfer line to transmit digital 
information to the image timer, and a power line to supply power to the camera. Similarly, 
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the cabling between the image timer and main control computer or any additional computers 
can have like capability. 

In still another aspect, the main control computer allows a user to access an 
object identifier and an associated time corresponding to the object, e.g., for use in a race- 
management system. For example, prior to the start of a race, the lanes within the camera 
field of view can be spatially referenced to a portion of the displayed image such that when a 
user points to that portion, both the lane number and any associated object, e.g., a race 
entrant, is available on the display monitor. Further, according to an additional aspect, the 
time and name of a particular object is automatically entered within a results window on the 
display monitor once a user so commands it. 

These and other aspects will become apparent in the following description, 
where the invention is described and illustrated in connection with certain preferred 
embodiments; however, it should be clear that various additions, subtractions, and 
modifications can be made by those skilled in the art without departing from the scope of the 

v 

invention. 

Brief Description of the Drawing s 

A more complete understanding of the invention may be obtained by reference 
to the drawings in which: 

FIGURE 1 schematically illustrates a system constructed in accordance with 
the invention for recording and displaying a time-sequential scene of bodies crossing a plane. 

FIGURE 1 A illustrates adaptive signal processing in a preferred camera used 
in the system of FIGURE 1 . 

FIGURE 2 illustrates how a system constructed according to the invention 
sequentially constructs a scene from discreetly sampled line objects. 

FIGURE 3 illustrates a collection of digital image frames forming a block of 
information, where each frame is marked with an associated time reference. 
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FIGURE 3 A illustrates a preferred embodiment of the invention in which 
particular frames are cropped from a sequence of frames. 

FIGURE 4 illustrates a preferred use of the invention in which a system 
constructed in accordance with the invention operates as a race-management system which 
views and records a race. 

FIGURE 5 illustrates a typical display of a racing scene generated by a system 
constructed in accordance with the invention. 

FIGURE 6 illustrates a system constructed in accordance with the invention 
having a virtual memory subsystem. 

FIGURE 7 shows a sequence of data corresponding to an illustrative first step 
in a compression of the sequence of digital image frames as provided by the invention. 

FIGURE 7A shows a second illustrative step in the compression of the data 
sequence shown in FIGURE 7. 

FIGURE 7B shows a third illustrative step in the compression of the data 
sequence shown in FIGURE 7A. 

FIGURE 8 schematically shows a first compression system constructed in 
accordance with the invention. 

FIGURE 8A schematically shows a second compression system constructed in 
accordance with the invention. 

FIGURE 8B shows representative data pixels generated in accord with the 
invention and within successive frames of a black and white camera. 

FIGURE 8C shows representative data pixels generated in accord with the 
invention and within successive frames of a color camera. 

FIGURES 8D-8G show successive and illustrative compression operations in 
accord with further features of the invention. 
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FIGURE 9 illustrates a preferred embodiment of the invention utilizing a 
virtual memory subsystem, multiple cameras, and multiple buffers within an image timer. 

5 FIGURES 9 A, 9B and 9C illustrate another multiple camera system with 

tunable timers in the cameras to mark frames. 

FIGURE 1 0 illustrates a display scene showing two scene sequences generated 
by two separate cameras. 

10 

FIGURES 1 1 and 1 1 A illustrate adaptive color palettization in a preferred 
embodiment of the system. 

Detailed Description of the Invention 

15 

FIGURE 1 illustrates a system 10 constructed in accordance with the 
invention for recording and displaying a sequence of bodies crossing a plane in space. The 
system includes a digital camera 12, an image timer 14, and a main control computer 16. 
With internal optics 15, the camera 12 views and images the line object 18 onto an array of 
20 detector elements 20, preferably a Line Scan Charge Coupled Device (LS-CCD). A camera 
processor 22 time-sequentially samples the image at the detector elements 20 and amplifies 
and digitizes the output signal at the gain controller 24 and AID converter 26, respectively. 
Each sampled image represents a frame of digital information at a unique moment in time. 

25 Each digital image frame is transmitted along a signal line 28, preferably a 

coaxial cable, to the buffer memory 30 of the image timer 14. The timer processor 32 marks 
each frame as it enters the image timer 1 4 with a time reference, preferably generated by the 
timer clock 34, by storing the time reference within the digital information of the frame. 
Thus each digital image frame stored in the buffer contains both the stored digital 

30 representation of the line object 1 8 and a unique time associated with it. In a preferred 
embodiment, the time reference for each frame is indicative of the time the camera 12 
captured the picture relative to the start of an external event. 

The buffer 30 stores the frames generated from the camera 12 until they 
35 accumulate to a preselected memory allocation, called a "block", after which the main control 
computer 16 transfers the block to its own internal memory 36 via a signal line 38. 
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The main control computer 16 has a central processor 40 that processes the 
blocks of information stored within the internal memory 36 so that the scene and time 
contents of a sequence of digital image frames are displayed on the display monitor 42. The 
central processor 40 also controls the automatic operation and memory management of the 
system 10, and responds to inputs at the keyboard 44 and mouse 45 so that a user can 
selectively command the display of any scene captured by the system 10, including a real- 
time display or previously recorded segments. More particularly, a user can access the 
unique times associated with any portion of the scene. 

In a preferred embodiment, commands to the digital camera 1 2 from the timer 
14 and the main control computer 16 are transmitted within the signal line 28, which is a 
single coaxial cable. The coaxial cable 28 additionally acts as a power control line to supply 
energy to the camera 12 so that the camera 12 can operate without a remote power source. 



With further reference and description of FIGURE 1, a three-dimensional 
orientation chart 50 is provided to facilitate a better understanding of the operation of the 
invention. The system 10 operates by sequentially capturing the image of the line object 1 8 
as viewed by the camera 12 at discrete moments in time. The line object 18 is typically only 

20 a fraction of a "scene" as observed by a person at the display monitor 42. That is, each line 
object 1 8 captured by the camera 12 sequentially forms part of a larger picture, or "scene", of 
the bodies moving by the field of view (FOV) of the camera 12. This FOV is essentially a 
plane in space, representable by the axes 52 and 54 of the chart 50, because the detector array 
20, and its conjugate line object 18, are practically one dimensional: the line object 18 has its 

25 long dimension along the axis 52 of the chart 50 and its short dimension (not shown) along 
the axis 56 perpendicular to the page of FIGURE 1 . The camera 12 focuses on the line object 
18 at a distance directed along the axis 56 from the camera to the object 18. 

Thus for example, FIGURE 2 illustrates an object 60 which is in motion along 
30 the axis 56 of chart 50', a 90° rotation of chart 50. The camera (not shown) is focused on the 
object 60 with a FOV substantially in the plane of axes 52 and 54. As each frame is captured, 
a portion of the object 60, i.e., a line object, is uniquely and spatially represented as a digital 
image frame. In FIGURE 2, the successive line objects captured by the system are 
illustratively shown on the object 60 as successive rectangles 62. For each of the line-objects 
35 62, the digital camera 12 correspondingly generates a frame by sampling the image of the line 
object according to the number of detector elements within the array 20. That is, each of the 
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line objects 62 is digitally segmented along its length (i.e., along the axis 52) into a digital 
image frame according to the sampling at the detector array 20 and transmitted at a moment 
in time to the image timer 14. In a real-time aspect, line object 64 represents the most 
recently captured frame and the remainder of the object 60 to the left of line object 64 has yet 
5 to be captured by the system. A scene or composite image of an object displayed on the 

computer 16 can look very much like the actual object passing by the FOV of the camera 12. 

As mentioned earlier, each digital image frame captured by the system 10 of 
FIGURE 1 is marked with a particular time reference from the camera and stored into blocks 
10 of information at the computer 16. The camera sends a special digital value to the timer 

which the timer recognizes as a start/end of a frame. The timer then marks the received frame 
with the associated time reference. 

FIGURE 3 illustrates more fully these operations. Each digital image frame 
15 70 captured by the system 10 includes an array of digital bytes corresponding to the signal 
detected by the activated elements of the array 20. When a frame enters the image timer 14, 
the timer processor stores the time associated with that frame in the last four bytes 72, 
thereby permanently associating each frame with a unique time. In FIGURE 3, time is shown 
increasing with the arrow 74; thus frames towards the left of FIGURE 3 are later in time than 
20 those on the right. 

FIGURE 3 also illustrates the collection of frames which form a block of 
information 76 utilized by the main control computer 16. According to a preferred 
embodiment, frames are organized into blocks of information that are 16k-bytes in size. The 
25 number of frames which make up the block 76 therefore depends upon the amount of 

information within each frame - which is a variable dependent upon the further features of the 
invention discussed below. 

FIGURE 3 A illustrates a particular feature of the invention which is enabled 
30 because of the unique time reference associated with each frame of FIGURE 3. In particular, 
according to a preferred embodiment, a user at the display monitor 42 of FIGURE 1 can 
select and crop selected frames 73 from the scene displayed without disrupting any of the 
information available in any other frames. A cropped portion 73 can be within a block 76, as 
illustrated in FIGURE 3 A, or some portion thereof (not shown). The address of the cropped 
35 frames are initially sent to a listing memory {a memory associated listing table) which holds 
their addresses until they are either erased permanently (which can occur by a "save " 
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operation) or re-inserted within the cropped sequence. This is particularly useful in a scene 
or race which has uninteresting segments that simply waste valuable memory. It also helps in 
the interpretation of a displayed scene because both the overall length of the scene as viewed 
from the monitor is decreased and the depth aspect of the displayed scene is improved. If, for 
5 instance, a first runner was several seconds ahead of a second runner, the recorded sequence 
between the runners can be cropped and the display on the screen appears as though no 
significant time between the runners exists. Of course, the time relationship of the runners 
remains accurate and when a user accesses the times associated with the runners, the time- 
discontinuity between the two will become apparent. 

10 

As discussed earlier, the most obvious use for a system constructed in 
accordance with the invention is directed towards race management. FIGURE 4 illustrates a 
system 10 in a configuration suitable to capture the motion of bodies crossing the finish line 
of a race. The system 10 is illustratively shown next to the race course 80 with the digital 

15 camera 12 located to view the plane representative of the finish line. The image timer 14 

receives digital image frames from the camera 12 at a frame rate selectable within the system 
1 0 and marks each frame with its associated time reference. The main control computer 1 6 
retrieves and stores the frames from the image timer 14 as blocks of information and displays 
the recorded scene on the display monitor 42. The computer 16 also allows a user, in the 

20 illustrated embodiment, to control certain features of the invention described below by the 
keyboard 44 and a computer mouse 45. 

Also shown in FIGURE 4 is a start sensor 84 which responds to the start of an 
external event, for example the start gun which signals that the race has begun, and which 
25 signals this time to the image timer 14. The timer clock 34 of FIGURE 1 is calibrated to this 
start signal and the timer processor 32 marks each of the frames entering the timer 14 with a 
time reference that is relative to the detected start time. 

A printer 86 can be installed with the system 10 to print selected scenes and 
30 information about the event recorded. 

FIGURE 5 illustrates a scene generated by the system 10 of a race containing 
four participants, exemplifying a typical display available on the monitor 42 (FIGURES 1 and 
4) and printer 86 (FIGURE 4). In FIGURE 5. the arrow 90 is in the time dimension and 
35 points to increasing time; while the arrow 92 refers to the spatial dimension corresponding to 
the spatial information within the digital image frames. This image can be zoomed in or out 
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on the monitor 42 (described in more detail below) by a user operating the computer 16. In 
the preferred embodiment, a user of the system 1 0 can select the resolution or image quality 
of a scene generated by the system 10 by adjusting any of three separate parameters. First, by 
adjusting the frame rate, i.e., the rate at which the camera captures each line object in the 
scene, the amount of resolution information available along the axis 90 changes. Secondly, 
by adjusting the camera density, i.e., by selectively activating or deactivating certain elements 
along the detector array 20, the resolution information available along the axis 92 changes. 
Third, by zooming in and out on the display monitor, the amount of displayed resolution 
changes in either or both of the axes 90 and 92. In general, the best display detail or 
resolution occurs when the user displays every frame recorded by the system 10 at a high 
frame rate and every pixel is activated on the detector array 20. 

However, other considerations must be made when adjusting these three 
parameters. First, it may seem intuitive that the highest frame rate available by the system is 
preferable in all instances. But if the frame rate is too fast, the objects can appear "fat" on the 
monitor 42, unless zoomed out. For example, if the race consists of runners traveling at 
about 20mph, approximately 500 frames per second makes the participants appear normal on 
the display 42 without zooming. If the entrants were instead tortoises, a much slower frame 
rate would be preferable (if they are slower by 1/100, for example, a frame rate of 5 Hz would 
be ideal). Another factor influenced by the frame rate is the energy available to the detectors 
within the camera. If the frame rate is too fast, the image could appear dim because the 
detectors did not have sufficient time to integrate the available light energy from the line 
object. This depends, of course, on the sensitivity of the detectors and the spectrum utilized 
by the camera. In the preferred embodiment, the detector array 20 of FIGURE 1 responds to 
visible light energy, and therefore requires more time to capture a particular image at dusk or 
twilight hours. If, on the other hand, the array 20 was constructed to respond to infrared 
energy, for example with HgCdTe material, the frame rate would be adjusted according to the 
temperature and speed of the objects. 

The invention thus provides an automatic gain control (AGC) mechanism to 
actively compensate for differing levels of light energy from the scene of interest. The 
camera processor 22 and gain controller 24 of FIGURE 1 programmably adjust the gain 
applied to the digital image frames transmitted from the camera to the image timer in real- 
time. The camera processor 22 responds to a command from the main control computer to 
raise or lower the gain in conjunction with the gain controller 24 by quantifying the digital 
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values within the digital image frames output from the AID converter 26 and modifying the 
sensitivity of the camera. 

For example, if the average digital value in a series of image frames is too 
low, the scene at the computer would appear dim. A command by the user at the main 
control computer to increase the gain appropriately improves the displayed screen contrast. 
A user can also select how fast the AGC control of the camera operates by adjusting the 
bandwidth of the gain control. A user effectively changes the AGC bandwidth by selecting 
the amount of time the camera 12 takes to quantify the average digital value used in setting 
the active gain. Although not generally practical for the majority of motion events, like races, 
the gain could theoretically vary for every frame generated by the camera by increasing the 
AGC bandwidth to a maximum setting. 

In a preferred embodiment of this aspect of the invention, the camera 12 
employs a novel signal conditioning circuit 24a, 24 (FIGURE 1 A) to improve the pixel data 
generated by the line sensor 20. As shown, the sensor output 20 is amplified by the gain 
controller 24 which generally selects one of a number of possible gain levels Lj based on 
camera operator or system operator commands as discussed above, and/or programmed 
microprocessor control based on current lighting conditions, scene light levels and the like, or 
a combination of program control and operator selection. The gain control 24 employs an 
initial preamplifier 24j which compares the sensor output level of each pixel to an offset 
voltage. In the prior art this has been generally a constant threshold voltage set at the factory 
based on sensor characteristics. However, in accordance with this aspect of the present 
invention, rather than a preset threshold, preamplifier 24j receives an offset voltage input 
from circuit (designated generally 24a) that is a variable function of the noise or dark current 
output of the CCD sensor array. The offset voltage is subtracted from the video output of the 
sensor by the preamplifier. The video signal is then amplified by one of several selectable 
gain circuits as before and the analog to digital converter 26 converts the amplified video 
signal into an 8 bit value. 

As shown in FIGURE 1 A, the CCD sensor 20 has a line of sensing pixels and 
also has one or several "dark pixels' 1 . These are regions of the array that have their charge 
accumulating and amplifying or gating structures fabricated identically to those of the active 
light sensing pixels, but they are masked so that no light reaches them. Their output thus 
represents the background noise level, due. for example, to penetrating or non-visible 
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radiation, heat, shot noise, and the like, and may thus be taken to represent the component of 
noise which appears in the output signal of each of the active sensing pixels. 

A CCD timing module 24j provides the clock signals for sending the sensor 
5 charge data to the initial preamplifier 24j, and also provides a dark pixel flag on line 24\ to a 
decision circuit 24^ during the time that the output from the dark pixels of sensor 20 appear 
in the data stream. Decision circuit 24^ receives the stream of 8-bit digital pixel values and 
holds the flagged dark pixel data, which it compares to a desired dark level. Decision module 
24^ then sends an UP or DOWN signal to a voltage integrator 24 m , the analog output of 

10 which provides the offset voltage to the preamplifier 24 j. The voltage integrator stores a 
voltage value that can be adjusted up or down in small increments by the UP and DOWN 
signals. As the offset voltage causes the dark pixel value to reach the desired level, the 
voltage integrator will hold the offset voltage constant if no UP or DOWN signals are 
received from the decision module 24^. The dark current offset module 24a thus serves as a 

15 closed loop feedback from the noise component of the output pixel data stream of A/D 

converter 26, to the gain control 24, and it operates to zero out noise contribution prior to the 
high gain video signal amplification of gain control 24. This results in enhanced image 
quality over a much greater temperature range, and assures that the preamplifier threshold 
level remains effective despite drift or component aging. Notably, the decision module 24^ 

20 operates on the digitized amplified dark current pixel value, and thus provides a highly 
accurate correction for current operating conditions. 

The circuit works on all frames, even during video capture. If the gain that is • 
selected is changed, then the offset voltage required may change also. The circuit will react 
25 to this and correct the offset voltage over the next few frames. 

As also shown in FIGURE 1 A, in addition to the amplified video signal with a 
gain level ... or L n on line 26a, the A/D converter 26 also receives a fine control 
adjustment signal on line 26b from D/A converter 24n. This is an analog voltage converted 
30 from an 8-bit control word which sets the range of the A/D converter 26 used to digitize the 
amplified video. As the value of the 8 bit control word provided to D/A converter 24n is 
reduced, lowering the voltage on line 26b, the A/D converter uses a smaller range. Thus, 
smaller inputs on line 26b represent larger portions of the range, and the image values are 
larger. In this way, lowering the D/A voltage on line 26b increases the apparent gain. 
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In a prototype embodiment, the coarse gain ranges are related by a factor of 
three, with relative gains of .5, 1 .5, 4.5 and 13.5, and the switching between gain levels is 
coordinated with corresponding changes in the control word on line 26a to avoid changes in 
magnitude. When switching from one gain Lj to another Lj+j , a corresponding change by a 
factor of three in the D/A signal on line 26b will result in a nearly identical image. For 
example, a coarse gain of 1.5 and a fine gain of 85 (that is, 255/3) corresponds to a coarse 
gain of 4.5 and a fine gain of 255. Thus, changes in fine gain are coordinated with operation 
of the gain selector, to smooth transitions between the coarse gain levels Lj, L2 - . . That is, 
as the module determines that light levels or operator commands will require a jump in gain 
level, the fine control word is decreased (if the gain is to go up) or increased (if down) so that 
when gain level is switched, the transition appears smooth. 

As previously mentioned, the resolution is modified in the spatial dimension 
92 of FIGURE 5 by changing the camera density which selectively activates certain detector 
elements on the detector array 20. Thus by decreasing the camera density by a two or four- 
factor, the resulting spatial resolution will similarly decrease by one-half or one-fourth, 
respectively. As more detectors are deactivated by decreasing the camera density, the amount 
of detail recorded along the axis 92 of FIGURE 5 decreases. This can be a desirable feature 
if high detail is not required for a particular scene since it significantly reduces the amount of 
information stored by the system 10. 

More particularly, the camera density is adjustable in a programmable fashion 
by a user at the computer 1 6 of FIGURE 1 , which transmits a signal to the camera processor 
22. The processor 22 thereafter selects only the appropriate data to send to the image timer 
14, corresponding to the commanded camera density that activates the particular detector 
elements within the array 20. 

With respect to displayed resolution, a user most readily adjusts the displayed 
image quality by zoom operations. Because each frame stored by the system contains a 
unique time reference, the process of zooming is easily attained by skipping or duplicating 
frames in the scene without compromising the critical time relationships within the scene. 
The mouse 45 of FIGURE 1 and 4 allows the user to point to particular objects on the scene 
and zoom either in or out by clicking the mouse, to thereby see more or less detail of the 
image, respectively. It is particularly useful when used in conjunction with the time-crop 
control discussed earlier whereby a user can zoom out and crop several sequences of "dead" 
time within a long race to shorten the stored file length . The zoom operation is available 
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along both directions of the displayed scene; that is, a zoom within the spatial and time 
dimensions may be made either concurrently or independently. Multiple zooms are centered 
on the display according to the selected point of interest. However, the information available 

« 

during zoom operations is constrained to the amount of information captured according to the 
5 frame rate and density settings of the digital camera; it does not generate new information. If 
the display resolution exceeds the available captured resolution, the displayed image can be 
smoothed by an appropriate interpolation scheme.. 

The amount of information captured by the system 1 0 is itself an important 
10 quantity. The LS-CCD detector element array 20 of FIGURE 1 is preferably 1024 elements 
long, which is commercially available. Once the detector array is sampled and digitized, each 
detector element activated has 8-bits, or a "byte", of information associated with the 
particular frame. In this configuration, each frame has 1024 bytes of information at the 
highest camera density. In a preferred embodiment, a block of information contains 16k- 
15 bytes of memory, and therefore sixteen frames form a block of information if every detector 
on the LS-CCD array is activated. 

However, if a user decreases the camera density by activating every other 
pixel along the array 20, the data amount within one frame is reduced by one-half, i.e., to 

20 512-bytes and the number of frames within a block of information increases to 32. This is an 
important feature because most computers are limited by processing speed and memory. If a 
long event is permanently recorded, at some point the amount of memory is exceeded. A 
virtual memory subsystem, or hard-disc drive, as described below in a preferred embodiment 
of the invention greatly adds to the amount of memory available for the system. 

25 Nevertheless, by reducing the camera density and frame rate, as well as the judicious 

cropping and data compression such as described below, the amount of digital information 
representing each frame and the rate at which data is transferred between the camera and the 
image timer can be greatly reduced. 

30 The data rate processed by the system 10 typically does not exceed lOMbits/s 

(e.g., corresponding to 1000 frames per second and 1024 active detector elements with 8-bits 
per element). Thus the system 10 is generally not subject to noise problems, time delays and 
processing constraints so that the distance between the camera 12 and timer 14 of FIGUREs 1 
and 4 can be at least one thousand feet in length. 
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Nevertheless, the amount of information stored and recorded can be large for a 
given event or series of events. At lOMbits/s, for example, every second of an event requires 
approximately one Mbyte of storage. In a preferred embodiment of the system 10, a virtual 
memory subsystem is included to accommodate the storage of data. FIGURE 6 illustrates a 
system 1 00 constructed in accordance with the teachings herein which includes a virtual 
memory subsystem, or hard-disc drive 102. 

The system 100 stores blocks of information into the virtual memory 
subsystem 102 when the blocks stored within the internal memory 36 for the main control 
computer 16 exceed a predetermined memory threshold. A user can select the threshold or 
rely on a default setting, e.g., 2-Mbytes of internal memory which can hold at least 125 
blocks of information. Functionally, the main control computer accesses each block in 
internal memory by an associated software pointer which has a zero value when the block is 
transferred to the virtual memory subsystem. 

Accordingly, the virtual memory subsystem can operate as an integral 
component of the system 100, and essentially transparent to a user. When a block of 
information is needed for processing by the main control computer 16, the block is 
transferred to internal memory 36 and an unneeded block transferred to the virtual memory 
subsystem 102. In this fashion, the system 100 can hold a scene of information which greatly 
exceeds the amount of RAM in the internal memory 36. In practical terms, the main internal 
memory 36 operates as a cache for the hard disc virtual memory 102. 

The semantics for accessing frames from any associated storage memory is 
straightforward. Since a determinable number of frames comprise each of the blocks of 
information, the frame number divided by the number of frames per block gives the correct 
block address, and the remainder gives the correct frame address within the selected block. If 
cropping is involved, memory is simply re-ordered into new blocks of frames, with a 
corresponding reallocation of frame addresses. 

Even with a very large memory capacity within the subsystem 1 02, it too can 
be exceeded when several motion events in a row are processed and stored, or long events are 
timed. The invention thus provides a compression system for reducing the amount of 
information needed in the blocks of information. The compression system is preferably 
available for use by a system constructed in accordance with the invention utilizing the 
virtual memory subsystem 1 02, which conveniently operates as a depository for the 
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compressed data. The data compression can be automatic or selectively chosen by a user, for 
example after recording a particular motion event. 

The compression system relies on the fact that each frame within the scene has 
the same background information within it. That is, if there is no motion activity within the 
FOV of the camera, each frame has practically the same information within it since the 
system constantly captures a single stationary line object; it is the motion of bodies crossing 
the FOV plane which generate a scene of interest. Thus, many frames stored for a given 
scene or motion event have redundant information. 

More particularly, the camera 12 of FIGURE 1 generates an 8-bit greyscale 
number, i.e., a number between a dynamic range having up to 256 shades of grey, for every 
active detector element in every frame. This corresponds to the amount of light energy 
within the image of the line object captured at that detector for a given frame. When a 
sequence of frames contains redundant information, the 8-bit numbers between successive 
frames are approximately equal. 

In a preferred embodiment, the first step taken by the compression system is to 
convert every 8-bit number in a selected sequence of digital image frames into a 5-bit number 
in the range 0-3 1 . Thus, an 8-bit number between the values 248-255 would be transformed 
to a 3 1; and a 8-bit number between 240-247 would be transformed to a 30; and so on. This 
compression process sacrifices the number of greyscales available within a picture (i.e. how 
"bright" a particular point on an object is represented by 32 numbers instead of 256 numbers) 
but saves a large amount of memory. It is worth noting that this is not a significant loss as 
some common VGA monitors have only 16 shades of grey available. 

FIGURE 7 illustratively shows a sequence of frames 112 containing seven 5- 
bit numbers with a possible digital value of "A" or "B", and an associated time reference 1 14 
in the last four bytes of the frame (in reality, there are 32 different values possible in this 
sequence, but "A" and "B" are used for ease of demonstration). Thus FIGURE 7 illustrates a 
sequence of frames after the initial transformation by the compression system from 8 -bit 
numbers to 5-bit numbers. In FIGURE 7 (including FIGURES 7A and 7B), each square 
represents a byte. As you can see, the redundant 5-bit numbers are very apparent and 
unnecessarily waste memory space. 
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FIGURE 7 A shows the next step of the compression system where each row 
of 5-bit numbers is reformatted by accumulating numbers of equal value within a row and 
then representing these accumulations as a "count" and a "value". The count corresponds to 
the number of equivalent 5-bit numbers in a series, and the "value" corresponds to the actual 
5 5-bit digital number. Thus, for example, the first row of sixteen 5-bit numbers "A" can be 
represented by a count "16" and a value "A". The second row has a count "2" having the 
value "A", followed by a count "7" having the value "B", and followed by a count "7" having 
a value of "A". This process continues until the information within every row of 5-bit 
numbers contains a progression of "counts" and "values", as FIGURE 7A illustrates. It is 
10 worth noting that if the "count" is less than or equal to "7", it is representable by a 3-bit 

number (the number "8" is reserved for an indication that the count will be represented by a 
separate byte). Thus in the second row of FIGURE 7A, each of the "counts" plus "values" 
can occupy the space of one 8-bit number. But if the count exceeds 7, the "count" and 
"value" numbers are each represented by a separate byte. 

15 

The last step of the compression system is shown in FIGURE 7B. The 
significantly trimmed rows of FIGURE 7 A are appended to each other to form the final and 
compressed representation of the original sequence of digital image frames, which now 
occupies a significantly smaller amount of memory. The time reference information 1 14 is 
20 kept unchanged, however. Note that provided m is less than n in this compression scheme, 
the file cannot exceed its original memory size. 

FIGURE 8 illustrates a compression system 120 constructed in accordance 
with the invention. In particular, FIGURE 8 shows three process actuators 121, 122, and 123 

25 which perform the steps of the operations described in FIGURE 7, 7 A, and 7B. A sequence 
of digital image frames enters the system 120 at a first data port 124. Process actuator 121 
converts each n-bit number within the sequence as a representative m-bit number, to form, for 
example, an array such as shown in FIGURE 7. Process actuator 122 reformats the array of 
rows of m-bit numbers into representative "counts" and "values" as for instance shown in 

30 FIGURE 7 A. Process actuator 123 again reformats the data to a sequential listing with the 
time reference information appended, such as shown in FIGURE 7B. Data thus compressed 
exits the compression system from a second data port 126 for transmission to a storage 
memory, e.g., the virtual memory subsystem. 

35 When a frame is required for processing by the main control computer, the 

compressed information is similarly uncompressed into a block of information before it is 
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processed for display and manipulation by a user of the system. Upon uncompression, 
however, only 32 greyscale numbers are available, not the original 256. 

As noted, there is a limit to the rate at which a computer, disk, or 
5 communication system can handle data. Typically, for example, the invention can generate 
visual scene information in excess of this data rate. This is particularly true if the scene 
information includes color information. Therefore, it may be necessary to compress scene 
information before it is stored by the computer, e.g., the main control computer 16 of 
FIGURE 1 . 

10 

Accordingly, the invention provides for another compression system 120a, 
FIGURE 8A, which achieves high image quality over the variable data rates generated by the 
camera, e.g. the camera 12 of FIGURE 1, as well as over the acceptable data rates handled by 
the computer. System 120a has three process actuators 121a, 122a, and 123a which perform 
15 the steps of the compression and quantizing operations described below with respect to 
FIGURES 8B-8G. 

FIGURE 8B shows representative data pixels 140 from a black and white 
camera, where each illustrated pixel has an associated 8-bit numerical value, 0-255, 
20 representing the amount of light (light intensity) received by that pixel during acquisition of 
that picture. FIGURE 8C, on the other hand, shows representative data pixels 142 from a 
color camera, wherein each pixel is represented by three numerical values 142a, 142b, and 
142c, which are assigned, respectively, to red, green, and blue light intensities. 

25 In either of FIGURES 8B and 8C, pixels are displaced horizontally along the 

time axis 144, and vertically along the spatial separation axis 146, such as described in 
connection with FIGURES 2 and 3. Each vertical array of data pixels is representative of the 
data acquired in one frame 148 of a scene. 

30 To compress the information in either of FIGURES 8B or 8C, system 120a 

performs a series of compression steps on the data. First, actuator 121a of system 120a 
subtracts pixels in one frame from the data pixels generated at the same spatial location in the 
previous frame, such as shown in FIGURE 8D. Accordingly, only those pixels whose values 
change will have non-zero values. This differencing operation of actuator 120a causes the 

35 relative distribution of pixel values to change: many more data pixels will be zero, or near 
zero, and those data pixels can be encoded in an efficient manner, such as described below. 
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The compression operation illustrated in FIGURE 8D can be performed 
similarly on color images by treating each color type as a single black and white image, such 
as shown FIGURE 8E. In this compression, actuator 121a of system 120a subtracts the red 
values of each frame from the red value at the same spatial location within adjacent frames. 
Similarly, actuator 1 2 1 a subtracts green values from the green values, and blue values from 
the blue values in adjacent frames. The encoded value of each resultant color pixel is then 
composed of three differences. 

The above-described subtraction operations can be thought of as a "predictor" 
function, namely that the next pixel in time is predicted to be the same as the previous frame. 
Therefore, actuator 121a produces a "predictor" to reduce the amount of data required to 
represent the information. It is necessary to store, or transmit, more information only when 
the given prediction is incorrect, for example, only if at least one of the B-A pixel entries is 
non-zero. 

This differencing predictor may be enhanced further to better predict the pixel 
values in adjacent frames. For example, FIGURE 8F illustrates a double differencing 
predictor technique in accord with the invention. Actuator 122a produces this predictor by 
computing a double difference whereby the difference of the preceding pixel pair is used to 
more accurately predict the current pixel. 

More particularly, FIGURE 8F shows four representative pixels A-D, which 
are sequential pixels in four adjacent frames. In this example, pixel D is the pixel being 
compressed by the actuator 122a. The predictor value of D, i.e., D pre( jictoD * s calculated as 
follows: 

^predictor =C + (B-A) 

This value is generated by the compressor at the camera, and by the 
decompressor when received from storage or along a communications line, from the already 
acquired values of A, B and C. 

Thereafter, the predictor error is encoded by actuator 122a as: 

^predictor ^predictor ■ 
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As with the simple differencing compression, it is only necessary to send this 
prediction error code if it is non-zero. 

5 The double differencing predictor of FIGURE 8F is easily extended to color 

images by treating each color portion of the pixel separately, similar to that described in 
FIGUREs 8B and 8C. That is, actuator 1 22a performs three similar operations described 
above for each of the three color components of the pixel. 

10 Actuator 123a of system 120a thereafter digitizes the image frames using an 8 

bit conversion process to produce image pixel values in the range of 0 to 255. These pixel 
values are preferably quantized by actuator 123a by converting the 8-bit number to a number 
with fewer bits, e.g., 5-bits, before displaying the information to a viewer. One method for 
quantizing the data is to truncate the pixel value and ignore any lower order bits, such as 

1 5 illustrated in FIGURE 8G. 

The invention also provides a more sophisticated quantizing process by 
detecting only those pixels values which differ from the predictor by an amount that is more 
than an amount T which represents a noise value and serves as the quantizing resolution step. 
20 Those pixels which oscillate across the quantizing step represent noise. Therefore, in accord 
with one embodiment of the invention, actuator 123 a generates those pixels influenced by 
noise as identical 5-bit truncated values, i.e., as zero difference pixels. An example of this 
enhanced quantizing process is illustrated in FIGURE 8H. Note that an upper quantizing step 
Tl and a different lower quantizing step T2 are shown, but a single step T may also be used. 

25 

Once the image data pixels are processed by the noise band quantizer and the 
predictor differencing processes above, the resulting set of pixels represent pixel differences. 
Since after applying the quantizer threshold T, a very large number of difference values will 
be zero, or nearly so, the data will be compressed from its original size. The differenced 
30 pixels may be further encoded, i.e., compressed, according to standard coding schemes, or 

according to the compression system such as described in connection with FIGURES 6 and 7. 

In a color system, it is possible to encode the three color differences as a single 
symbol. Such an encoding may take advantage of correlated behaviour of the colors. 
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The compression system 120a of FIGURE 8 A may be augmented or 
supplemented with structure of the process actuators 121, 122, and 123 of FIGURE 8, as 
appropriate, to perform additional or supplemental compression steps. Thus, as in 
compression system 120 of FIGURE 8, a sequence of digital image frames entering the 
system 120a at a first data port 124a may be converted by process actuator 121a from n-bit 
numbers to m-bit numbers, to form, for example, an array such as shown in FIGURE 7. 
Process actuator 122a thereafter reformats the array of rows of m-bit numbers into 
representative "counts" and "values" as for instance shown in FIGURE 7A. Process actuator 
123a again reformats the data to a sequential listing with the time reference information 
appended, such as shown in FIGURE 7B. Data thus compressed exits the compression 
system from a second data port 126a for transmission to a storage memory, e.g., the virtual 
memory subsystem. 

One important feature or advantage of the aforementioned compression 
system 120a is its real-time tunable compression capability. A single hard-wired 
compression circuit would force the system to pick a mid-point in image quality, thereby 
lowering the image quality even if the bandwidth were adequate for the image stream; or 
further limiting the speed even in cases where image quality is not a concern. Tunable 
compression, in accord with a preferred aspect of the invention, performs this trade-off in 
real-time and only as necessary. 

■ 

This is accomplished as follows. A buffer is provided in the camera or in the 
compression module 120a for receiving the input line image data stream 124a, and a buffer 
monitor indicates how full the buffer is, e.g. ten percent, fifty percent, eighty percent. The 
type of compression implemented in module 120a is then selected based on the backlog of 
uncompressed image data in the buffer. At one extreme, there is no compression, or 
compression by a method such as first described above, which is lossless, or very nearlv so. 
At the other end is a compression method which allows a large "noise margin" in the 
quantizer, and reduces the bandwidth of the data stream enormously, at the expense of image 
quality. Since this noise margin can be set instantaneously in the compression module, the 
quality of the image can be adjusted such that it is maximized for a given data rate. For 
example, using the original wire interface of the prototype system which can move 
500KBytes per second, raw 16-bit-per-pixel image data would only go at a rate of 250 lines 
per second for a 1000-pixel line. In order to send lines faster than this, one must compress 
the image data in real time. How much compression to use depends on the line rate needed 
and the inherent "compressibility" of the data (a scene that is not changing often, for example, 
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is much more compressible since the frame-to-frame differences are very small). The size of 
the buffer is selected so that the system can temporarily get "behind" in its compression of the 
data and only switch to a more drastic compression if the buffer continues to fall behind. 

5 The system sets the tunable parameters based on the expected continuous data 

rate rather than overreacting to small spikes in the data rate caused by changes in the scene. 
One way of achieving this is for the system to implement a decision rule that it will use no 
compression until the data fills 10% of the buffer. At that point a small amount of 
compression is added. When the buffer reaches some critical state (say, 75% full) the system 
10 may resort to drastic measures, such as dropping every other frame or every other pixel. 

When used as an event recording system, such as at the finish line of a race, 
the line scan camera system of the present invention may be operated to record and resolve 
motions one thousandth of a second apart. However, in such a setting, it is of the utmost 

15 importance not to lose whole sections of the image. The goal is to always completely utilize 
the 500KB/s bandwidth of the wire while keeping the image as high a quality as possible. 
The 500KB/s not only represents the bandwidth of a wire interface to the camera, but is very 
close to the best performance of modem disk drives. Thus, even in order to capture the image 
direct to disk, this data rate should be maintained. By locating the compression module 1 20a 

20 at or in the camera, none of the processing power of the system computer 16 is diverted to the 
compression process. 

The foregoing "tunable" adaptive method of compression is highly effective 
for applicant's line images, and it is unlike compression used in many other video 

25 applications in that it sacrifices the quality of regions of the image for the quality of the 

edges. By contrast, most video or image compression algorithms (including Discrete Cosine 
Transform (DCT) or JPEG tend to smooth the edges in favor of the regions. In applicant's 
event camera application, it is the edges which typically determine which object is in front of 
which. Thus, the compression retains or enhances frame to frame edge discrimination at high 

30 data rates. Experience with a simple prototype implementation of this algorithm which has a 
theoretical upper limit of compression of five has established that in a well lit scene it is 
common to get real-time data compression ratios in the range of 3.5 to 4.5, with little 
noticeable effect on image quality. 

35 In accordance with yet another aspect of a preferred embodiment of the 

present invention, a color line scan event camera as described above or, more preferably, an 
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operator controlled computer 16 at a monitoring site, generates a small color palette to which 
it fits the color values of the pixels in its sequence of frames, so that each 15-bit RGB pixel 
value is represented by one of no more than 256 colors in the palette. In accordance with this 
aspect of the invention, rather than using a preset palette, or a two-pass procedure wherein all 
pixel data colors in every frame are reviewed and a palette is selected which most accurately 
represents the range of colors, the palette composition is adaptive, and its entries are selected 
as the frame data is being processed. 

This aspect of the invention addresses the difficulties of color display in the 
uniquely time sensitive environment of the present invention. In general, compressed video 
data from the camera is stored, with time-marked information, so that it is accessible modulo 
an inherent latency time due to the network communications, storage access, and generally 
also decompression processing. To make these delays sequential with a two-scan 
palettization process, or even to display retrieved and delayed excerpts with a small fixed 
palette, would be restrictive. In accordance with the present invention, video data, either 
directly from the camera or as it is retrieved from storage, is palettized for 256 color display 
by generating a color assignment table "on the fly". The palettization process introduces no 
observable delay, and each frame sequence begins with nearly perfect color fidelity. Later 
frames are then forced to fit the small color palette adapt ively derived from the initial frames. 

FIGURE 1 1 illustrates this aspect of the invention. Each line or "frame" Fl, 
. Fm generated by the color camera has n pixels specified by a five bit word for each of the 
red, green, and blue components, the 15 source bits making 2 15 = 32K possible colors. This 
data can be directly used on a video display operating in the so-called "high color" mode of a 
video card capable of simultaneous display of 32K colors. However, to be useful for 
interactive examination of the image - e.g. for the operator to call up the image, inspect 
frames and time markings while viewing material on a display - it is generally preferable that 
the raw data be mapped to a destination image containing fewer bits per pixel. Many laptops 
and other PC's, for example, are operated in the 256 color mode. 

It is desirable that the source image colors appear similar to the image formed 
with only the small palette of destination image colors. However the usual processes for such 
mapping involve either a time consuming run through the entire source image to identify the 
range of colors and pick a reduced set of representative colors that are sufficiently close, or 
involve mapping all colors to a predetermined, fixed palette. In the latter case, while the 
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transformation to the color format of the destination image is quick, the colors may appear 
quite different from the actual view, leading to a misleading or unsettling appearance. 

This problem is solved in accordance with a color palettization aspect of the 
5 present invention by a palettizer 2 1 0, which may for example be implemented in software in 
the terminal 16 (FIGURE 1), that receives frames Fi of source image data and converts their 
RGB valued pixels Pi to pixel values P'j having a small set of color values Ci in a destination 
image frame F'j. The values Ci are adaptively determined and built into a color table 220 as 
the conversion proceeds. 

10 

FIGURE 1 1 A illustrates this pixel color processing. The palettizer 210 is 
activated when an operator calls up a sequence of frames, typically specified by the timing 
markers which index all images. The color look up table 220 is initially empty. Palettizer 
210 inspects the pixel values PI, P2 . . . Pn of the first frame Fl. initially choosing the value 
15 of PI as the first color CI to add to the table 220 and assign to corresponding pixel P'j of the 
destination frame. Next a determination is made whether the value of the second pixel P2 of 
the frame is sufficiently close to CI to represent it by the same color. Closeness may be 
defined, for example, by a distance function operating on the fifteen bit RGB coordinates, 
such as a sum of the differences in RGB values, or sum of squares of differences. 

20 

Other "distance" functions may be used which have been empirically found to 
reflect cognitive judgments of "closeness" in human color perception, to assure, for example, 
that a low intensity pure blue is never mapped to a low intensity red. 

25 Continuing with the description of FIGURE 1 1A, if P2 is close to PI, then the 

color value P'2 is also set to CI. Otherwise, a new color C2 is entered in the table 220 and 
the value of P'2 in the destination image frame is set equal to C2. Thereafter, each pixel 
value Pi is inspected in order to determine if it is close to one of the already assigned colors 
Ci-k 0 for some value k 0 <i, and, if so, Pi is mapped to the table color 

30 Cj - k Q . Otherwise, that is, if Pi is not close to any color, then provided the table 220 has not 
already been filled, the value of pixel Pi is used as the next color entry C in the table and is 
assigned to the pixel P'j in the destination frame. Otherwise, that is, if the table 220 is full 
but Pi is not close to (e.g., within 5 of) any Ci, then a different mapping is employed. The 
pixel Pi is simply assigned that value Cj which is closer than the other colors of the table. At 

35 this stage in the processing, the palettizer operates on the pixels in sequence, and moves on to 
the next frame. 
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In this manner, a set of up to 256 colors are established for a color mapping 
palette, in which the palette initially matches the source colors very closely. After a number 
of frames have been processed and the table 220 is full, subsequently appearing colors will be 
"forced" into the existing palette and the image color will be less faithful. However, in the 
context of a fixed view line scanning camera, certain portions of the image, such as the track 
and landscape background of a race finish line, will remain unchanged throughout all frames, 
and will appear in accurate colors having been coded as initial entries in the adaptive color 
palette. Furthermore, when the color mapping is only used to call up and examine a few 
seconds' duration of line scan image data, the 256 colors will suffice to fully and accurately 
represent all frames. 

The palettization steps, involving simple "closeness" measures on an incoming 
digital word pixel value against a small set of color value words from the table, is extremely 
fast, and palettization may proceed in real time as the frames are viewed. 

With reference to FIGURE 9, a system 101 constructed in accordance with the 
invention also shows an additional computer 104 and additional buffers 106 and 108 within 
the image timer 110. These additions are not at all required for the operation of the virtual 
memory subsystem 102, or any other component described above, but rather are useful in 
further features of the invention. 

In particular, the computer 104 allows an additional user to access and analyze 
portions of previously recorded segments that are stored onto the hard-disc 102. Similar to 
the main computer 16 of FIGURE 1, blocks of information are loaded into the internal 
memory 116 before processing and displaying the information. All of the features available 
at the main control computer 1 6 are also available at the additional computer 1 04, thereby 
providing a convenient forum for other management of the data processed during a sequence 
of motion events, e.g., races already recorded. Of course the main control computer 1 6 can 
also operate to review previously recorded segments of any prior motion event - and even 
during the activities of a current motion event - or operate in a real-time mode and display the 
current motion event as captured by the camera 12. 

A plurality of computers, like the computer 104, are similarly attached to the 
virtual memory subsystem if other users wish to simultaneously access the data stored on the 
hard disc. A computer suitable for use within the system described includes common IBM 
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personai computers or compatibles with an 8086 processor, a VGA video adapter, and 640K 
of RAM. 

The buffers 106 and 108 within the image timer 110 of FIGURE 9 illustrate 
5 another important feature of the invention, namely the addition of a plurality of digital 
cameras to the system 10 of FIGURE 1 . In some circumstances, it is desirable to view the 
motion of bodies crossing the plane in space from two or more vantage points. For example, 
in a race it is very possible in a close heat that one race participant blocks the view of a single 
camera with respect to another race participant. To correct this potential problem, one or 
10 more additional digital cameras, like the first one, can generate additional sequences of 

digital image frames of the line object of interest. Preferably, each additional camera views 
substantially the same plane in space, e.g., two cameras on either side of a race track 
exemplifies the use. 

15 Since the buffers for every camera are within the image timer, a single clock 

112 provides the time reference for all frames entering the timer. The timer processor 114 
can thus mark each frame with a calibrated time reference thereby permitting each camera to 
operate asynchronously. That is, any of the plurality of cameras can have any selected frame 
rate or density, and the image timer continues to accurately mark each frame as it enters. 

20 

In a preferred system, rather than designing a multi-buffer timekeeping 
module for receiving and marking frames from a plurality of different cameras, the system 
employs cameras having tunable timers that are maintained accurate enough to mark each 
frame with an accurate time before sending it to storage or the central processor. The tunable 

25 timers are not themselves extremely accurate, and they may be implemented with simple 
oscillator circuits and dividers to form a local clock. However, each tunable camera 
periodically communicates with a precision timer and after initially establishing synchronous 
time, periodically re-tunes its clock rate to maintain synchronicity. Thus, in a system 
employing such cameras, each frame generated by a camera is marked with an "absolute 11 

30 time marking. 

One embodiment of such a system 200 is illustrated in FIGURES 9 A and 9B. 
The system includes a plurality of line scan cameras of which one is a primary camera C \ and 
the rest are secondary or slave cameras Cj The primary camera C| contains a precision time 
35 source Tl, and an imaging portion V] consisting of a line scan sensor assembly and 
associated video components substantially as described above. Each slave camera C; 
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contains a tunable time source T2 as described below, with an imaging portion Vj identical 
to that of the first camera. All cameras connect over cabling, so that the secondary cameras 
may communicate, either directly or via a central control computer, with the precision time 
source of camera C\. It will be understood in the discussion below that while Cj has been 
indicated as containing a precision time source, to which each slave camera refers, the system 
may also consist entirely of slave cameras. In that event, camera C] is simply another slave 
camera, and a separate precision time source is provided, such as that of image timer 14 of 
FIGURE 1, to which all the slave cameras periodically synchronize their clocks. The process 
of synchronizing includes specifying a time offset for each camera C, in a manner known in 
the art for dispersed ranging or time recording systems, to compensate for the delays in 
sending and receiving time check messages along the cabling referencing the source timer Tl 
(i.e. conduction delays, switching delays, and network message handling delays), and this 
initial offset correction will accordingly be assumed in the discussion that follows. 

The local timer T2 of each slave camera is shown in FIGURE 9B, and consists 
of a local oscillator LO together with one or more intermediate frequency stages SI, S2, S3 
and a clock synthesis or counter stage S4. The intermediate frequency stages typically are 
formed by simple divide-by-n circuits - e.g. special logic arrays - while the clock synthesis 
stage comprises a circuit of one or more adders, counters and the like which constitute a timer 
using selected ones of the IF signals. The local oscillator LO may be any locally available 
frequency source, such as a 16 MHz clock of the camera's microprocessor controller, a 
separate timing chip or the like. Since the IF divider stages simply divide the local oscillator 
frequency, the resulting local timer is accurate to within a tolerance corresponding to that of 
the oscillator. 

As shown in FIjGURE 9B, one of the intermediate frequency clock stages, 
illustratively SI, referred to herein as the "prescaler" stage, contains two divider circuits with 
closely related divisors m and m+k, where k is a small number such as 1 or 2, so that 
depending which divider is used, it puts out a clock pulse train of frequency f7m or f7(m+k), f 
being the local oscillator frequency. The divisors m and m+k are selected so that one 
frequency f/m results in a faster-than-actual time clock rate, while the other f7(m+k) results in 
a slower-than-actual clock operation. 

For use in a line scan event timing camera, it is preferable that the timer 
resolve imaged events with a resolution of approximately one millisecond or better. One 
representative embodiment having such resolution is implemented using a 16 MHz local 
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oscillator, a single frequency divider or prescaler stage SI, and a 32-bit counter. The 
prescaler for normal clock operation might divide the oscillator frequency by a number n to 
achieve an approximately accurate clock. In accordance with the present invention the 
numbers m and k referred to above are selected to be n-1 and 2, respectively, so that the 
5 prescaler contains a divide by (n-1) and a divide by (n+1) circuit. One of these (n-1) results 
in a fast IF clock "F", while the other (n+1) produces a slow IF clock "S". A selector operates 
at some rate controlled by the camera processing circuits, which exchange timing information 
with the precision time source and select one divisor or the other depending whether the 
clock time is presently early or late. The rate of these special cycles and the direction of 
10 timing error determines the apparent speed of the local oscillator at any instant in time, and 
once synchronicity is established, maintain the absolute timing error within preset limits. 

For example, a 16 MHz oscillator would nominally be divided by 125 to form 
a 128KHz clock value. If the prescaler uses a divide by 124 cycle every millisecond, then the 

15 nominal 128 KHz clock rate will increase to 128.0079125 KHz, a 61 parts per million 

increase. Likewise, if the prescaler uses a divide by 126 cycle every millisecond, the clock 
rate will decrease by 61 parts per million. Thus, by changing the amount of time during 
which N+1 or N-1 cycle prescaling is performed by the prescaler, the clock may be advanced 
or retarded by several milliseconds per minute, and by retiming at much shorter intervals, the 

20 clock time may be maintained with the necessary accuracy. 

By using three divisors 124, 125 and 126, the clock may be run at a slow, 
normal or fast rate. In this case, the normal divisor 125 may be used by default, until the 
comparator determines it is necessary to accelerate or retard the clock. FIGURE 9C shows 
25 such a local clock with three discrete timing rates for the phase-locked tracking of the 
precision source. 

This capability allows the camera to adjust its clock until it matches the clock 
of the reference timer. Once the camera has adjusted its clock well enough, it can use its own 
30 local timer value to mark time on the image frames. Periodically, the camera checks actual 
time and sets the prescaler timing to advance or retard the local clock to keep in 
synchronicity. 

In this manner, the non-precision local oscillator in each secondary camera 
35 provides a continuously tuned local clock that, although apparently speeding up or slowing 
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down by as much as one tenth of one percent, changes its direction of error frequently and 
always tracks the precision timer within one, or even one-tenth of a, millisecond. 

FIGURE 10 illustrates a typical display of a race having tvvo separate cameras 
on opposing sides of the track. This image is available to any of the computers discussed 
herein, including the additional computer 104, provided at least two cameras are integrated 
into a system constructed in accordance with the invention. 

In a preferred embodiment, the direction of any displayed sequence of digital 
image frames forming a scene is reversible by re-ordering the sequence of frames. Thus for 
example, both of the scenes displayed within FIGURE 10 can be reversed if selected in 
preference by a user of the computer. By operation of a mouse therefore, the separate views 
of the identical participants within the two scenes can appear to have a motion in the same 
direction. 

The system described herein has other features available to users at any 
connected computer. Prior to recording an motion event, for example, a selectable part of the 
displayed scene can be uniquely associated with an object identifier. Thus along the spatial 
(vertical) domain of the screen, one or more lanes on a race course - and in particular the 
runner within - can be associated with a particular portion of the screen. A user can, for 
example, point to that portion and acquire the information relating to that lane, like the 
entrant's name. Furthermore, because every portion displayed on the screen has a unique 
time reference associated with it, a user can similarly access and display the time associated 
with a selected portion on the screen and display its results. In operation, for example, a user 
can select the lane of the winner in a race to see the person's name; and, more particularly, a 
user can select the first portion of the winner's body crossing the finish line to display the 
associated win time. In a preferred embodiment, both the object identifier and the time 
associated with any portion of the scene on display are automatically entered in a results table 
if selected by the user. 

Any display, showing a scene, race results, or any other information selectable 
by the user can be printed at any time by an appropriate print command selected at that 
computer. 
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The term "greyscale" as used herein shall include not just classic greyscale 
values but also can include an equivalent color representation (e.g., either a single value or a 
series of values representing different colors). 

Appendix I is an object code listing of a program for carrying out many of the 
functions described herein. This program is executable on any Intel X86 based personal 
computer system under MS-DOS. 

It is accordingly intended that all matters contained in the above description or 
shown in the accompanying drawings be interpreted as illustrative rather than in a limiting 
way. 

It is also understood that the following claims are intended to claim all of the 
specific and generic features of the invention as described herein, and all the statements of the 
scope of the invention. 
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Claims 

1 . An event recording camera system for forming a time sequence of 
images, comprising: 

A. a first digital camera which generates a first sequence of digital image 
frames, each frame of said first sequence representing a first view that is captured by said first 
digital camera at a first moment in time and imaged onto a first array of detector elements; 

B. a second digital camera which generates a second sequence of digital 
image frames, each frame of said second sequence representing a second view that is 
captured by said second digital camera at a second moment in time and imaged onto a second 
array o f detector e lement s ; 

C. a precision timer associated with said first digital camera, and a timer 
processing means which marks frames of said first sequence of digital image frames with a 
time reference indicative of said first moment in time, 

D. a tunable timer in said second digital camera, said tunable timer being 
operative in at least a faster mode and in a slower mode, and timer processing means which 
marks frames of said second sequence with a time reference representative of said second 
moment in time as determined by the tunable timer; and 

E. mode selection means for accessing the precision timer and selecting a 
mode having a speed to maintain said tunable timer synchronous with said precision timer, so 
that said first and second sequences of digital image frames are marked with a common time 
reference. 

2. A system according to claim 1, wherein said tunable timer includes a 
prescaler with different dividers for providing different frequency bases, the different 
frequency bases determining modes of different speed. 

3. A system according to claim 1, wherein said precision timer is in said 
first digital camera. 



I 
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4. An event recording digital camera including an optical imaging 
element for forming a light image of a visual field, and an image sensing element for 
generating a digital image frame from the light image, timing means in said camera for 
generating time signals at tunable discrete rates, and means responsive to communication 
5 with a precision time source for selecting a said rate so as to maintain synchronization of the 
timing means with the precision time source, said digital camera further including time 
processing means for marking frames generated by said camera with their time of generation 
as determined by said timing means. 

10 5. A digital camera according to claim 4, wherein the image sensing 

element is a line sensor. 

6. An event recording digital camera including an optical image forming 
element and an image sensing element for generating digital image frames, a tunable timer 

15 and means for periodically varying the rate at which said tunable timer operates so as to 
maintain it in synchrony with a precision time source, and means for marking the digital 
image frames with time generated by said tunable timer. 

7. A digital camera according to claim 6, wherein said sensing element is 

20 a line sensor which generates frames at a rate substantially over one hundred frames per 

second, and said tunable timer is phase locked to said precision time source to maintain a 

~» * 

timing accuracy of greater than approximately \0' D sec. 

8. A digital camera including an optical imaging element for forming a 
25 visual image of a field, an image sensing element for generating a multi-pixel electrical 

output from the visual image, and circuit means in said camera for processing said multi- 
pixel electrical output to provide a video stream including a digital value for each image 
pixel, wherein said circuit means senses a digitized dark current signal from a dark current 
pixel of said image sensing element and offsets the electrical output of each image pixel by a 
30 value corresponding to said dark current signal, thereby producing a low noise video stream 
of enhanced image quality. 

9. A digital camera according to claim 8, wherein said video stream is an 
amplified digitized video stream, and said circuit means provides an offset value to a 

35 preamplifier that amplifies the electrical output generated by the sensing element, said offset 
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value being selected to maintain amplified digitized dark current below a preset threshold in 
said video stream. 

10. An event recording video camera comprising 
optical means for imaging a visual field 

a line sensor for converting a fixed line portion of the imaged visual 
field into electrical signals representative of a line of pixels of the image of said field, the line 
of pixels forming a frame representative of said fixed portion at an instant in time 

circuit means for processing said electrical signals to form a digital 
output stream containing digitized pixel values of a sequence of frames at a high scanning 
rate 

a buffer for storing said digital output stream before transmission, and 
compression means for compressing data in said buffer to reduce 
bandwidth thereof before it is transmitted. 

1 1 . An event recording camera according to claim 10, further comprising 
selection means responsive to contents of the buffer for selecting a compression mode, said 
selection means selecting a first mode of compression when said buffer is relatively empty, 
and selecting a second mode of compression when said buffer is relatively full, said second 
mode compressing more than said first mode. 

12. A video camera according to claim 1 1 , wherein one of said 
compression modes includes line to line differencing compression. 

13. A video camera according to claim 1 1, wherein said compression 
means includes noise band quantization means, for compressing two digitized pixels that 
differ only by a threshold noise amount to a single compressed value. 

14. A video camera according to claim 11, wherein said compression 
mode includes frame to frame predictive coding. 

15. A video camera according to claim 1 1 , wherein said pixel values 
include color values and said compression mode compresses values for each color separately. 

16. A video display processing module for processing a data stream of 
video data for display, such data including successive frames of light information, each frame 
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comprising pixels having light color values, wherein the color values may represent a large 
number of colors, said display module including means for determining a palette of colors 
consisting of a smaller predetermined maximum number of coded color values, wherein the 
means for determining includes 

5 

i) first means for assigning successive coded color values to said 
palette based upon color values of pixels in the data stream, said first means including 
comparator means for comparing color value of an incoming pixel with coded color values if 
any previously assigned to said palette, and 

10 

ii) means responsive to said comparator means for replacing the 
color value of an incoming pixel with a coded color value from the palette when said 
comparator means determines a match, and otherwise assigning the color value of said 
incoming pixel to the palette as a further coded color value, until said palette contains said 

15 maximum number of coded color values, 

said palette initially being non-full, and said means for assigning 
operating to assign said small number of colors to all pixels of successive frames as it fills 
said palette so a video image represented by said data stream may be viewed immediately 
20 without awaiting inspection of subsequent frames. 

17. A video display processing module according to claim 16, wherein 
said comparator means operates when said palette is full, to determine a color from the 
palette which is closest to color value of an incoming pixel. 

25 

18. A video display processing module according to claim 16, wherein 
said palette comprises a look up table holding 256 color entries. 

19. A video display processing module according to claim 16, wherein 

30 said comparator means determines a distance between incoming pixel colors and colors of the 
palette. 

20. A video display processing module according to claim 19, wherein 
said distance is a cognitive distance measure corresponding to human color perception. 

35 
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21 . A color normalizer for determining a small palette of fewer than n 
colors and assigning a color from the palette to each pixel of a stream of color video data 
having plural frames of pixels, wherein each pixel is specified by multibit RGB components, 
thereby to produce a multicolor stream with fewer colors suitable for display on a monitor, 
such color normalizer comprising 

first means inspecting RGB values of incoming pixels in a digital data 
stream and for adaptively adding colors to the palette until it is filled with no more than n 
colors selected in accordance with the RGB values of said incoming pixels, and for 
simultaneously assigning colors of this palette to said pixels, and 

second means operative once the palette is filled, for assigning the 
closest color of the palette to each incoming pixel, 

whereby each pixel is assigned one of said fewer than n colors for 
immediate interactive n-color computer display of said stream by an operator. 

22. A color normalizer for determining a small palette of fewer than n 
colors and assigning a color from the palette to each pixel of a stream of color video data 
containing plural frames of pixels, wherein the video data includes views of changing events 
against a fixed environment, and said normalizer adaptively forms a table of said n colors on 
the fly based upon colors of initial frames in said color video data stream, and then assigns 
the n colors from the table to subsequent frames of said stream. 

23. A compression system for compressing a sequence of digital data 
frames, wherein each of said digital data frames forms a column of n-bit greyscale numbers 
representative of luminance values detected at pixels of a line sensor on which a fixed line 
portion of a scene is imaged in a video camera, and wherein said sequence forms an array of 
rows of n-bit numbers, such compression system comprising: 

A. greyscale conversion means, for converting each of said digital data frames 
into a column of m-bit greyscale numbers, wherein the integer m is less than n, thereby 
forming an array of rows of m-bit numbers; 

B. a counter means, which provides a count of the number of occurrences of 
equivalent m-bit greyscale numbers in adjacent frames in each row of said array of rows of 
m-bit numbers; 
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C. a row reformatting means, for reformatting each row of said array of rows 
of m-bit numbers such that every row sequence of equivalent m-bit numbers is represented by 
one m-bit number value and the count thereof; and 

D. a memory reformatting means, for storing each reformatted row of said 
array of rows sequentially; 

whereby said compression system reduces the memory required to store said sequence of 
digital data frames representing the fixed line portion of the imaged visual field. 

24. The compression system of claim 23 , wherein said row reformatting 
means reformats each row of said array of rows of m-bit numbers such that said one m-bit 
number and the count thereof are represented by one n-bit number. 

25. The compression system of claim 23, wherein said row reformatting 
means reformats each row of said array of rows of m-bit numbers such that said count thereof 
is represented by one n-bit number; 

26. The compression system of claim 23 , further comprising compression 
accuracy control means, for selectively controlling the accuracy of the information stored by 
said compression system by selectively altering said integer m. 
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